Neural network models can be utilized for various purposes, such as providing output that is utilized to control (directly or indirectly) one or more operations of a robot. As one example, a neural network model may be utilized to determine a classification of an object in a raw image. For instance, the raw image may be applied as input to the neural network model and one or more value(s) may be generated over the model based on the applied input. Those values may indicate the object class(es) to which the object belongs and one or more operations of a robot may be controlled based on the indicated object class(es). For example, if the object is of a set of one or more first classes, the robot may attempt to grasp the object. As another example, if the object is of a set of one or more second classes, the robot may adjust its trajectory to avoid the object.